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ABSTRACT 


Having established clear evidence for a form of learning that is transformational and 
non-linear (Powell & Shklov, 1992), the authors herewith present a non-linear way to 
consider single cell frequency data in social-science settings. This paper describes the 
procedure in detail and gives several instances of its application in diverse settings. 
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The Problem 


These two researchers have, for some time, been investigating non-linear approaches 
to observational data. The initial problem arose because they were trying to find some way to 
examine, in multiple-choice tests, the patterns in which students’ answers (whether right or 
wrong) change with learning. 

As background to this problem there are two possibilities. The first possibility is that 
all wrong answers are blind guesses and that students will change from any wrong answer to 
the right one with appropriate instruction. In this case the current practice of considering only 
right answers when observing student behavior changes for quality control would be entirely 
appropriate. There would be no meaningful information to be found among the wrong 
answers. 

The second possibility, which comes from Powell’s teaching experience and from 
Piaget’s many years of clinical observation (see: Flavell, 1963), is that answers change in 
some sort of sequential order that is dependent upon how students interpret the test questions. 
Their interpretations, in turn, could be dependent upon their current style of thinking and 
their current depth of intellectual maturity. 

In this case, learning would display transformational properties (systematic answer 
changes) in addition to cumulative properties (increasing numbers of correct answers). There 
could be considerable meaningful information in all answers, not merely the right ones. 

To look at change, it is necessary to give the same test more than once. When this is 
done, it becomes possible to compare the answers selected upon the first administration to 
those selected on the second administration and build frequency tables that summarize these 
selections. In some cases, the students may select the same answer on both administrations, 
or they may change from one answer to another within each item. 
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If the first possibility (that students guess blindly until they know the answer) is 
correct, the typical observation should be changes to the right answer from any wrong one 
followed by systematic repetition of the right answer over time. No other combination of 
selection should be statistically significant beyond a chance level. 

If the second possibility is correct, then a much more complex pattern of answer 
stability and answer change should become evident as students transform their thinking with 
increased maturity. 

From a statistical perspective the problem is to determine the cumulative probability 
of a single observed frequency in a cross-tabulated frequency table with either P > 0.95 or 
P < 0.05 when the table is not diagonally symettrical. 

The Approach 

In a four-option multiple-choice test there will be a frequency matrix for each item 
containing 16 cells The one direction may be used to represent the responses from the first 
administration of the test and the other direction to represent the responses from the second 
administration. In this case each cell will be the frequency of the event of joint selection for 
each pre-post response pair. Those students who, for some reason, omitted making a choice 
at either of two administrations may be dropped from consideration because these data will 
not add to the information about the dynamics of answering. 

With these frequency tables, there are four pieces of information that can be 
considered firm for each cell frequency. These are: 

The number of times the members of the sample chose both members of the pair 
of responses being considered (the observed frequency), 
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The number of times this group chose the first member of the pair on the first 
administration, (the column total) 

The number of times this group of students chose the second member of this pair 
on the second administration (the row total), and 

The total number of observations in the entire frequency table (table total). 

In mathematical terms, the values being considered are the cell frequency, its 
associated row and column sums, and the table sum. 

When these matrices are displayed, they look something like the typical contingency 
table, except that the concern here is with each cell instead of the overall departure of the 
entire matrix from a x 2 distribution. These matrices are not diagonally symmetrical because 
events such as A => is not equivalent to B => A and either event is mutually exclusive to the 
other. 


Insert Table 1 about here 


Suppose, as illustrated in Table l.a, that we have a joint choice of 1 1, a first variable 
frequency of 15, and a second variable frequency of 19. If the group size is 23, the expected 
joint choice frequency is (15 x 19)/23 = 12.39. In this case, 11 is a lesser value than this 
expectation but may not be enough below this expected value to conclude that choosing one 
option implies the rejection of the other. If the group size is 90, as illustrated in Table l.b, the 
expected frequency is (15 x 19)/90 = 3.17. Is 11 enough larger than this expectation to 
indicate an outlier? 

After investigation of the literature and numerous simulations of existing proposals to 
find a way to put a probability value on the cell in question with known marginal sums and 
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any total frequency, it became evident that a more sophisticated approach was required for an 
effective analytical solution to this interpretive problem. One such resolution is reported in 
this paper. 

Because a single-dimensional array can be treated as a binomial distribution, these 
researchers tried using the multinomial distribution for the two dimensional arrays being 
studied. They developed an adaptation of this approach that involves collapsing these tables 
to two-by-two tables around each cell of concern. They then calculated the probability of 
observing that frequency or less with the other three values held constant. It is the purpose of 
this paper to provide the details of this procedure, with examples from different types of 
qualitative research. 

The Procedure 

Step 1. 

This procedure begins with an m x n matrix of frequencies Oij as follows: 

Oi„ Ri 

Oln Ri 

Omn Rm 

C n N 
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Step 2 


The next step is to collapse this matrix around any cell Oij . This step produces a 2 x 2 
matrix, with the observed frequency of Oij being o, of the sort: 


0 

/ 

R'i 

where: 

ll 

jo 

1 

0 

g 

h 

R'2 


R '2 = JV-R'i , 

CT 

C'2 
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g = C'i - 0 , 
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II 
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\ 

O 




and 

h = N - (0 +f+g ). 


In words, /is the frequency of the remainder of the row, g is the frequency of the remainder 
of the column and h is the residual frequency of the total table. Tables 2 and 3 give a 
numerical example of these first two steps in this procedure, beginning with the 4x4 
frequency matrix, such as: 


Insert Table 2 about here 


By choosing to look at O 22 (with a frequency of o = 33; the repeated choice of the 
correct answer as shown in the box in Table 2) the collapsed matrix becomes: 


Insert Table 3 about here 


In order to determine the probability that o would be 33 by chance alone, it is 
necessary to find two cumulative probabilities. First, it is necessary to find the cumulative 
probability for the range of all possible values in that cell with the three marginal values kept 
constant. Second, there is the need to find the cumulative probability for all single tables that 
have a value of 33 or less. 
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To achieve this end, it is necessary to find the smallest possible value and the largest 
possible value that o can achieve within these marginal constraints. 

Since these values are all frequencies, and the least possible frequency is zero (0), the 
least possible value that o can possess is when either the frequency value of o or ofh is zero 
(0). Similarly, the greatest possible value that cell Oij can contain occurs when either the 
frequency of f or of g is zero (0). 

Mathematically: 

O(min) = 0, if O < h\ Otherwise 0 ( 1 , 1 , 11 ) = o - h(4) 

0 ( max) = the lesser of Rd and CT (5) 

In the present case, the minimum value is 0 (zero) and the maximum value is 46. It is 
theoretically possible to have a matrix wherein the minimum possible value is not 0 (zero). 

Step 3 

The multinomial function is applied to the range of all possible values from the 
minimum (0) to the maximum (46), adding these partial probabilities to find the total 
possible probability (P r ) of events within these constrained conditions. The equation 
becomes: 


0(max) 

Pr =Z 

of fu\ gf hu\ 
t — O(min) 


(pifu ipifu ( pf) 8 u ipf) h u 


( 6 ) 


In this process, the value of u is the incremental value across the range from 0( m in) to 
O(max). This means that the values of Ow and h„ will increase by one (1) and of f, and g u will 
decrease by one (1) until the range has been completed. 



The values of pi, etc. are constant. These are the probabilities for each of the four 


cells that are determined by constraining the calculations using the marginal totals. These 
constraints are imposed because we are assuming that the marginal proportions are the best 
estimates available for the actual distribution of these particular choices in the total sample. 
Therefore these probabilities are the product of the joint proportions for each cell as 


follows: 

pi = RT C'i/N 2 (7) 

P2 = RT C'i/N 2 (8) 

p 3 = RT C' 1 /N 2 (9) 

p 4 = RT CT IN 2 (10) 

Also: 

f u = RT - Ou (11) 

gu = CT - ou (12) 

and h u = N - (o u +fu + gu) (13) 


In this particular example, the values of pi, pi, pi, and p\ for o are 0.182, 0.254, 

0.236, and 0.328 respectively, which adds to 1.000, for these four probabilities. 

Step 4 

The next step is to find the cumulative probability for the observed frequency or less 
(Pc ), this time iterating from 0( m in) to o using the same equation (6) and control parameters 
(7- 13) in step 3, as follows: 


o 

Pc , 7 . N! . 7 , • (pi)T (pi) f u (pi) 8 u (P4) h u 

Ou! fu! gu! n u ! 

C — O(min) 


(14) 
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The difference between these two calculations being that equation (6) increments to 
O(max) from O(min) whereas equation (14) stops incrimination at the observed value of o. 

Step 5 

The desired cumulative probability for the observed cell frequency can now be 
determined. The cumulative probability for all possible events within these constrained 
conditions is P t from equation (6) and the cumulative probability within these same 
constraints for the observed frequency or less is P t from equation (14). Thus the desired 
probability is the ratio of these two partial probabilities, or: 

Po =P C /Pt. (14)/(6) (15) 

In this example, the value of P c is 0.0059 and of P t is also 0.0059, so the probability 
of observing 33 choices of answer o with 46 and 48 overall choices and 110 observations, is 
Po = 1.000. It appears, from these actual calculations, that the remaining 13 possible values 
(from 34 through 46) contribute less than one part in 20,000 of the total possible probability, 
hence the resulting probability is 1.000 when these partial probabilities are rounded to the 
fourth decimal place. 

This probability represents the repeated choice of the right answer to this question at 

a particular age range (166 - 170 months). In this example thirty percent (33/1 10).of all 

answers are repeatedly right, while about two thirds (( 33 + 33 / (46 + 48) or 66/ 94) of 

those who chose this answer on either administration chose it on both administrations. By 

chance alone the expectation would have been only 20.07 such choices within the constraints 

of the marginal values If their selections had been entirely random, this expectation would 

have dropped to 1/16 of the table total, or 6.875 events 
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The Results; A First Look 


The next step is to consider the entire matrix given in Table 2 to see what this 
procedure can reveal. Here are the results of this one group of interactions across time within 
this one question. Table 4 gives the observed frequencies, expected frequencies and 
probabilities of all 16 interactions. It presents the frequencies that produced significantly high 
probabilities in enlarged bold print and the significantly low ones in reduced font size. 


Insert Table 4 about here 


Seven of the 16 interactions are significant; three high and four low. All the high ones 
are in the main diagonal, which means that there is a strong tendency in this age group for 
repeating the initial answer, without regard as to whether it is right or wrong. 

There are also some very serious problems with these outcomes. One of the highest 
frequencies among the off-diagonal cells is O 21 contains 8 joint selections but is classified as 
a significantly low probability of occurrence. The same event can be seen, to a greater or 
lesser degree for all the significantly low selection frequencies. The most frequent changes 
from one answer to another seem to be significantly low whenever the same two responses 
are significantly high in their repeated choices. 

Perhaps this problem comes the mutually exclusive nature of multiple -choice test 
answering. In this case, these low probabilities could be statistical artifacts. To test this 
hypothesis, a second-order calculation was performed. These probabilities were recalculated 
after making the main diagonal elements zero (0). Table 5 reports these results: 


Insert Table 5 about here 
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The results of the analysis of the frequencies from Table 2 can now be summarized in 
Table 6. 


Insert Table 6 about here 


The four significantly low probabilities disappeared in this second order analysis. The 
two low ones with relatively high frequencies compared with other off-diagonal elements 
became significantly high. In addition, one that was almost significant also became 
significantly high. There are now 6 of the 16 cells in the table that are significantly high, 
accounting for 78 of the 110 responses (more than 70 %), and none significantly low. This 
outcome compares with 33 stable responses (30 %) to the right answer. By regarding 
statistically significant probabilities to be psychologically meaningful, these frequencies 
now account for nearly two and a half times as much information as the stable right answers 
by themselves provide (78 / 33). 

The conclusion that can be drawn from these results is that, once this second order 
analysis has been applied, this procedure seems to effectively identify the transformative- 
answer patterns that appear to be present upon visual examination of the original 4x4 table. 
With such encouraging results, the final step is to apply this procedure to a larger data set to 
observe whether it performs as well in a more general applied context. The balance of this 
paper is devoted to giving three examples of its application as a demonstration of its 
effectiveness, or lack thereof, for the non-linear, non-traditional analysis of frequency tables 
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Applications of this procedure 


1. Reconsidering wrong answers 

The first presentation is a restatement of the results from the data for which this 
procedure was developed as reported in Powell and Shklov (1992). It comes from two 
administrations of the same test to 2810 subjects ranging in age from less than eight years of 
age to more than eighteen, with a five-month interval between administrations. The entire 
sample was then subdivided into 27 age levels of five months each. The reported age in 
months comes from the subjects’ ages at the first administration. The example we used above 
was age level 166 - 170 (median 168). 

These data come from item 18 of Gorham’s (1956) The Proverbs Test. This item 

reads: 

QUICKLY COME, QUICKLY GO (Easy come, Easy go). 

a) Always coming and going and never satisfied (median age 14). 

b) What you get easily does not mean much to you (right answer). 

c) Always do things on time (median age 8). 

d) Most people do as they please and go as they please (median age 10). 

A more detailed discussion of the interpretations characterizing these answers is 
given elsewhere (Powell, 1977, Powell and Shklov, 1992) in the literature. In the earlier 
study, this test was given to 550 elementary school children (grades 3 through 8) and then a 
representative sample of them was interviewed to determine the reasoning behind their 
answer selection. 

In brief, alternative “c” was chosen most frequently by 8 year-olds. Their reasoning 

showed that they had interpreted “Quickly come ...” as it represented their personal physical 

activities in their third grade classroom. They had many changes of activity in a school day, 
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all of which required that they “finish on time.” In Piaget’s terms they would appear to be 
still egocentric and intuitive in their thinking. 

Alternative “d” was chosen most frequently by 10 year-olds, with their reasoning 
reports suggesting that they were looking for the best literal interpretation of the proverb. 
This change in reasoning would suggest that they were no longer egocentric and were 
thinking concretely instead. 

Next in age sequence was alternative “a”, most commonly chosen by 14 year-olds. 
Their reasoning seemed to suggest that they were moving out of literal and concrete thinking 
into more figurative interpretations. 

The right answer, “b,” was chosen most commonly by 16 year-olds, suggesting that 
either they had reached formal operations or were answering from memory because of 
personal familiarity with this proverb. 

This present report uses the data from a later study. The purpose of this later study 
was to try to replicate the sequence just presented and to determine whether the changes of 
answer selection coincided with the possibility that a developmental sequence might be 
present. 

The present discussion provides the outcomes from the two orders of analysis 
combined into one table for clarity and convenience. Table 7 gives this consolidation with 
the numbers in larger font and in bold face being the significantly high frequencies drawn 
from this procedure. The numbers in smaller font and the light box are the significantly low 
frequencies. No significantly low frequencies occurred in the main diagonal. 

Insert Table 7 here 
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The most obvious observation is the huge discrepancy between the number of cells of 
significantly high frequency to those of significantly low frequency (200 to 3). Similarly the 
number of significantly high frequency cells to the total number of cells (200 to 432) shows a 
considerable departure from random expectations, suggesting that these observations are 
reflecting some underlying systematic aspect of behavior. In the random case the number of 
significantly high cells and the number of significantly low cells should be about equal. 

Considering the amount of recovered information is also revealing. Knowing that the 
correct answer to this question is “B,” adds additional insight. Nearly all of the stable choices 
of this answer between the two administrations of the test are significantly high, suggesting a 
stronger tendency to repeat this choice than any other. However this stability, which is 
similar to test-retest reliability, accounts for only 23.8 % (669 to 2810) of all the answers and 
32 % (669 to 2112) of all the significantly high frequencies. 

In contrast the total of all significantly high frequencies to all answers is 75.2 % 

(2112 to 2810). This procedure appears to be recovering a great deal more systematic 
answering than the repeated correct answers alone can supply. The results of the entire group 
are similar to the observations made with age level 166 - 170 months reported earlier. 

Also illuminating is the patterns of stability and change that can be observed. One 
way of defining a sequence is to consider two significantly high frequencies in chronological 
order initiate a sequence and two non-significant or significantly low cells break the 
sequence. Figure 1 gives these observations. These results are rearranged to show the evident 
sequential patterns. 

Insert Figure 1 here 
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In the first place, it may be noted that the sequences of significantly stable choices 
(c => c, d => d, ...) tend to follow the same sequential pattern found in the earlier study. In 
addition, the median points of the changes from one answer to another (c => d, d => a, ...) 
tend to fall between the solid commencement ages for the stable repeated choices of this next 
sequential alternative. Thus answers appear to display a replicating developmental sequence. 

Of particular interest is the change from the right answer “b” to the highest sequential 
wrong answer “a” with the more mature students. This reverse trend seems to reflect a move 
away from convergent thinking such as a multiple-choice test typically requires. 

It is also interesting to note that these data seem to suggest there may be at least three 
levels of achievement present in those who leave school as part of the main-stream. There are 
those who have consolidated upon literal thinking (alternative “d”), those who have either 
completed formal operations or who can give the right answer from memory (alternative 
“b”), and those who either remained at the level of personalized thinking or gone beyond 
formal operations (alternative “a”). There are also a large number of students who are in the 
transition stages among these several developmental levels when they graduate. 

Of further interest is the observation that there appears to be a second pathway as 
well. In this case the order seems to be the reverse of the main stream, going from the right 
answer “b” at a young age toward either literal thinking “d” or non-logical thinking “c” by 
the time that they leave school. Some of these students on this second pathway appear to 
leave school before they graduate. It was found that those students who withdrew from 
school between the two administrations most commonly chose either “c” or “d.” 

The appearance of more than one developmental pathway is itself an interesting 
psychological observation. It begs the question, “How many such pathways can be found 
using this procedure?” Further research will be needed to answer this question. The 
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observation that this reverse pattern may be detectable at as young an age as ten years is 
encouraging. In combination these observations seem to indicate that using right answers as a 
criterion for academic success may be necessary but might be an insufficient measure of 
academic progress. The students’ reasoning appears to determine the answers selected. It 
may be, as the interview data seems to suggest, that these transformational patterns among 
the answers could provide a window upon otherwise invisible learning processes. 

This information is unavailable when tests are scored “right-wrong.” Wrong-answer 
data is eliminated during scoring. Do these observations invalidate using this approach for 
test evaluation or educational quality assurance? Should we be using the information from all 
student answers instead? 

2. Teachers ’ attitudes toward their profession 

This second example being reported here is a fragment of a much larger study being 
conducted by the Consortium for Cross-Cultural Research in Education based at the School 
of Education of the University of Michigan in Ann Arbor. The study upon which this report 
is based was conducted in twelve countries, Allen Menlo being the coordinator. This 
fragment was prepared jointly between Powell and Collett (1990). 

Secondary- school teachers were asked indicate, within brainstorming sessions, their 
sources for enthusiasm and discouragement toward their on-the-job experiences as teachers. 
Their comments were collected on wall charts, discussed, and then summarized. 

Using a coding system, the written records of these discussions were reduced to seven 
categories, each of which could be designated as either as a source for enthusiasm (E) or 
discouragement (D). These categories were also classified as being controllable (C) by these 
teachers or beyond their immediate control (U) The categories were: 
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Self (C) 


Students (C) 

Others (C) 

Self (U) 

Students (U) 

Climate (U) 

Workload (U) 

By looking at only three countries (Canada, Japan, and the U.S.A.) the multinomial 
procedure was used to consider similarities and differences among these countries on these 
fourteen categories. When the multinomial procedure is applied to the table the results were: 

Insert Table 8 here 

Two trends are observable. First, each country shows some differences from the 
other, with Canada and the U.S.A. more alike with each having eight of the fourteen 
significant probabilities in the identical direction. The directions of the other six were also 
equivalent. Japan differed from the other two countries by showing greater polarization and 
having an opposite view of the uncontrollable aspects of self and the classroom environment. 

Second, there appears to be a trend wherein the sources for enthusiasm come from the 
elements of teaching over which teachers have control, and of discouragement from those 
elements over which they have little control. Although these conclusions contain no 
surprises, the procedure seems to identify quite clearly both general trends and international 
differences among these data. 

3. Psychological characteristics of contrasting groups 
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In the third study reported here, Carney (1992) presents some simulated data designed 
to illustrate a clustering technique to aid in interpreting counseling sessions. In this paper 
twelve people, all of whom are adult children of alcoholic families, are contrasted. Six are in 
recovery and six are in denial. Table 8 gives the frequencies of responses in seven different 
categories for each of these six clients. 

Insert Table 9 about here 

These categories were compared as to topic, such that “living in a storm,” for those in 
recovery, was seen to be similar to “sudden, nasty mood swings,” for those in denial. Six of 
the seven could be contrasted in this way and their frequencies of response rank-ordered. The 
resulting p (rho) was - 0.49 which is negative but not significant. The multinomial 
probabilities are also given in Table 9. 

Only one of these six pairs of content variables showed statistically significant 
contrast. However, four of the remaining five show a trend toward a similar polarity. It would 
appear that people in denial, and in recovery tend to display opposite psychological attitudes 
based solely upon the frequencies of things about which they are willing to talk during 
counseling sessions. 

Conclusions and Implications 

In this paper three different contexts have been presented, each one having in 
common frequencies of some sort. In each case the multinomial procedure has either 
supported the conclusions that would be expected from the situations that these data 
represent, or they have cast new light on situations where more traditional statistical 
procedures appear to be insufficient if not inappropriate. 
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This statistical procedure seems to be a promising new tool for researchers who are 
uncomfortable with more traditional techniques or are finding that such techniques do not 
penetrate their data as deeply as they might wish. The large increase in explained variability 
that sometimes has been observed herein holds promise for drawing more detail from data 
than more traditional statistical techniques may supply. Whenever there may be reason to 
suspect a high complexity among the events contained in frequency-table data, this procedure 
could prove helpful. 
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Figure labels 


1. Change Patterns for the Entire Item 18 (N = 2810). 
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a. 

Option X All other Totals 
Options 

Option Y 11 4 15 

All Others 8 0 8 


Totals 19 4 23 


Expected value for 11 is 12.39 


1 
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Table 2 


An Example of the Procedure 
Using Multiple-choice Test Data 


Posttest 

Frequency of answers on Pretest 
A B C D Totals 

A 

11 

5 

3 

9 

28 

B 

8 

33 

2 

5 

48 

C 

3 

2 

0 

2 

7 

D 

7 

6 

3 

11 

27 

Totals 

29 

46 

8 

27 

110 


Source: Data for age levels 166 - 170 months from Table 4, Powell & Shklov (1992). 
The correct answer to this question is “B.” 
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Table 3 


The Results of Collapsing 
Table 1 Around Cell O 22 


Posttest 

Columno 

Columns 1+3+4 

Totals 

Rowo 

33 

15 

48 

ROWS 1+3+4 

13 

49 

62 

Totals 

46 

64 

110 
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Table 4 


Probabilities of the Observed Frequencies 
from Table 1: A First Look 


Row 1: 
Observed 
Expected 
p values 

11 

7.38 

0.997 

5 

11.71 

0.002 

3 

2.04 

0.888 

9 

6.87 

0.907 

Row 2: 

Observed 

8 

33 

2 

5 

Expected 

12.65 

20.07 

3.49 

11.78 

p values 

0.034 

1.000 

0.235 

0.002 

Row 3: 

Observed 

3 

2 

0 

2 

Expected 

1.85 

2.92 

0.51 

1.72 

p values 

0.923 

0.376 

0.580 

0.771 

Row 4: 

Observed 

7 

6 

3 

11 

Expected 

7.12 

11.29 

1.96 

6.62 

p values 

0.584 

0.014 

0.901 

0.993 


Note: Probabilities where p < .95 are shown in increased font and bold face; 
while probabilities p > 0.05 are shown in reduced font size. 
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Table 5 


Second Order Analysis of Table 


with the Frequencies in Main Diagonal Removed 


Row 1: 

Observed 
Expected 
p values 

*** 
* ** 
* *** 

5 

1.18 

0.846 

3 

2.47 

0.805 

9 

4.95 

0.998 

Row 2: 

Observed 

8 

*** 

2 

5 

Expected 

4.91 

* ** 

2.8 

4.36 

p values 

0.989 

* *** 

0.624 

0.778 

Row 3: 

Observed 

3 

2 

*** 

2 

Expected 

2.29 

1.65 

* ** 

2.04 

p values 

0.851 

0.796 

* *** 

0.674 

Row 4: 

Observed 

7 

6 

3 

*** 

Expected 

5.24 

3.78 

4.65 

* 

p values 

0.922 

0.968 

0.839 

* *** 
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Table 6 


Summary of the Results 
of Both Orders of Analysis From Table 1 


Posttest 

Frequency of answers on Pretest 
A B CD 

Totals 

A 

■eqj 

5 

3 


28 

B 

□ 

OD 

2 

5 

48 

C 

3 

2 

0 

2 

7 

D 

7 

□ 

3 

□ 

27 

Totals 

29 

46 

8 

27 

110 


Note: Frequencies with a probability p < 0.95 are shown in the boxes. There were no 
low-probability events remaining in the analysis of this table. 
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Table 7 


The Results of the Analysis of the two Administrations 
of Item 18 from The Proverbs Test 


Significance of Test-Retest Within-Item Responses 



A 

A 

A 

A 

B 

B 

B 

B 

C 

C 

C 

C 

D 

D 

D 

D 

Age 

Mean- 

Ages 

to 

to 

to 

to 

to 

to 

to 

to 

to 

to 

to 

to 

to 

to 

to 

to 

Group 

ingful 


A 

B 

C 

D 

A 

B 

C 

D 

A 

B 

C 

D 

A 

B 

C 

D 

Totals 

Count 

<96 

1 

0 

1 

2 

0 

0 

0 

1 

0 

1 

3 

3 

1 

3 

5 

3 

24 

16 

98 

0 

1 

1 

1 

2 

0 

4 

1 

1 

1 

2 

2 

1 

1 

2 

11 

31 

17 

103 

2 

1 

0 

0 

1 

0 

0 

2 

2 

3 

5 

3 

5 

3 

4 

7 

38 

17 

108 

15 

6 

4 

9 

4 

5 

3 

2 

6 

0 

5 

6 

5 

7 

5 

19 

101 

81 

113 

11 

1 

3 

5 

2 

0 

1 

3 

4 

0 

4 

7 

3 

4 

5 

10 

63 

27 

118 

5 

2 

7 

18 

5 

4 

3 

3 

5 

1 

8 

3 

7 

3 

8 

10 

92 

58 

123 

11 

3 

0 

13 

5 

4 

2 

2 

4 

0 

6 

8 

10 

8 

6 

17 

99 

71 

128 

16 

6 

2 

10 

7 

4 

3 

4 

6 

4 

2 

4 

10 

3 

5 

11 

97 

31 

133 

15 

5 

1 

12 

5 

13 

4 

6 

4 

0 

3 

5 

9 

8 

4 

21 

115 

90 

138 

9 

5 

3 

16 

4 

13 

1 

0 

3 

4 

4 

3 

13 

7 

3 

7 

95 

50 

143 

16 

10 

3 

8 

8 

15 

0 

6 

1 

6 

3 

3 

16 

5 

5 

12 

117 

93 

148 

10 

8 

4 

5 

5 

16 

3 

5 

1 

3 

2 

3 

7 

5 

4 

9 

90 

63 

153 

11 

3 

2 

11 

5 

14 

1 

5 

4 

2 

0 

3 

4 

5 

3 

14 

87 

67 

158 

10 

8 

0 

5 

9 

20 

2 

6 

4 

2 

5 

2 

10 

6 

4 

5 

98 

57 

163 

10 

5 

1 

4 

4 

16 

1 

4 

3 

1 

1 

1 

6 

1 

4 

11 

73 

63 



8 

3 

7 

5 

33 

2 

6 

3 

2 


3 

9 

5 

2 

11 

110 

78 

173 

16 

17 

3 

7 

6 

60 

5 

8 

1 

3 

3 

2 

13 

17 

5 

22 

188 

144 

178 

15 

17 

3 

14 

5 

54 

4 

9 

2 

7 

3 

3 

12 

20 

3 

22 

193 

153 

183 

9 

20 

2 

10 

15 

70 

2 

10 

4 

5 

1 

2 

10 

20 

3 

18 

201 

163 

188 

16 

14 

0 

6 

15 

63 

4 

5 

1 

4 

2 

1 

5 

17 

6 

18 

177 

157 

193 

15 

14 

0 

7 

10 

56 

8 

5 

0 

3 

0 

4 

8 

12 

1 

17 

160 

151 

198 

7 

18 

4 

10 

9 

48 

1 

9 

1 

2 

1 

3 

6 

9 

2 

14 

144 

114 

203 

9 

11 

1 

4 

12 

38 

2 

6 

0 

3 

0 

1 

7 

7 

3 

13 

117 

92 

208 

3 

8 

0 

1 

7 

38 

4 

6 

0 

0 

0 

1 

3 

9 

1 

7 

88 

80 

213 

9 

5 

1 

2 

6 

38 

1 

3 

0 

1 

1 

0 

1 

7 

2 

7 

84 

77 

218 

3 

4 

1 

6 

4 

16 

5 

3 

2 

2 

1 

0 

3 

2 

1 

5 

58 

39 

>220 

3 

5 

1 

1 

5 

35 

2 

4 

0 

1 

1 

0 

3 

4 

0 

5 

70 

63 

Totals 

258 

205 

51 

194 

165 

673 

68 

124 

62 

61 

66 

76 

187 

198 

96 

326 

2810 

2112 

Totals 

196 

158 

0 

151 

124 

669 

34 

85 

20 

13 

50 

42 

130 

136 

60 

244 

2112 



Source: Composite of Tables 4 & 5 from Powell & Shklov (1992). 

Age level 168 was used for the calculations in the example. 
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Table 8 


Attribution Study 
Basic Data and Probabilities 


Attribution 


u . s . 

A. 



Japan 


Canada 

Target 

Enthus- 

Discour- 

Enthus- 

Discour- 

Enthus- 

Discour- 


iasm 

agement 

iasm 

agement 

iasm 

agement 


/ 

p 

/ 

P 

/ 

P 

/ P 

/ 

P 

/ P 

Controllable: 










Self 

46 

1.00 

3 

0.00 

52 

0.99 

33 0.00 

76 

1.00 

10 0.00 

Students) 

48 

1.00 

16 

0.00 

128 

1.00 

26 0.00 

89 

1.00 

11 0.00 

Others 

2 

0.93 

1 

0.33 

6 

0.97 

2 0.13 

5 

0.85 

3 0.38 

Uncontrollable: 










Self 

7 

0.98 

2 

0.10 

n 

0.00 

41 1.00 

12 

0.63 

11 0.53 

Students 

11 

0.46 

12 

0.70 

51 

0.00 

104 1.00 

18 

0.00 

441.00 

Climate 

63 

0.00 

99 

1.00 

157 

0.96 

138 0.02 

78 

0.00 

1611.00 

Workload 

18 

0.00 

55 

1.00 

13 

0.00 

87 1.00 

19 

0.00 

481.00 


Totals 196 188 418 431 298 288 

Grand Total 1819 


BOLD: Frequencies with p < 0.95. 
Standard size: Frequencies not significant, 
small: Frequencies with p > 0.05. 
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Table 9 


Frequencies of Responses to Various Topics 
During Counseling Sessions 
by Adult Children of Alcoholics 
in Recovery of Denial 

In Recovery In Denial 


Topic 

of discussion 

Frequency 
of references 

Probab- 

ility 

Topic Frequency Probab 

of discussion of references ility 

1. Living in a storm 

« I 

0.9961 

6. Sudden, nasty mood 






swings 

23 

0.009 

2. Crazy making 

56 

0.884 

5. Unfair preferences 

29 

0.172 

3. Over- achieving 

55 

0.117 

1. Don’t do what dad did 

45 

0.921 

4. Frozen feelings 

52 

0.584 

3. Emotionally left home 

33 

0.512 

5. Survivor guilt 

47 

0.159 

2. Constant criticism 

38 

0.892 

6. Not belonging 

38 

0.146 

4. Physical violence 

32 

0.905 
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Main Stream Pattern Drop-out Patterns 

C C D D A A B* B* A + B* B* A A D D C 

Ages to to to to to to to to to to to to to to to to 



B* A A D D C C 


Literal Ego- 
cen 



School leaving age (16 years old). 

*1' Significant selection frequencies ip < 0.945). 


u => 


Non-significant frequencies (sequence continuing). 
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0 0 


Some of the identified developmental pathways. 
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